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Today’s  Talk 


Purpose  &  scope  of  the  survey 

Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 
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Understanding  the  State  of  Measurement 


Careful  &  well  executed  use  of  measurement  &  analysis 

•  Is  a  well  accepted  tenet  in  many  fields  of  endeavor 

•  Including  of  course  CMM I 

Basic  aims 

•  To  inform  management  &  technical  decisions  based  on  empirical  evidence 

•  &  to  judge  the  results  of  those  decisions  once  made 

But,  how  well,  and  how  frequently,  are  measurement  practices  put  into 
effect  in  our  own  field? 
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Surveys  &  Benchmarking 


Benchmarking:  The  current  state 

•  Some  professional  &  consulting  organizations  maintain  repositories  they 
use  for  establishing  benchmarks  &  facilitating  benchmarking  activities 

•  However,  their  measures  &  measurement  definitions  differ  in  many  ways 

•  In  that  sense,  one  cannot  speak  confidently  about  “industry  standards” 

•  Which  is  why  the  SEI  has  launched  the  Performance  Benchmarking 
Consortium  {as  described  at  last  year’s  CMMI  Technology  Conference} 

The  state  of  the  practice  surveys 

•  Aim  to  provide  data  that's  not  yet  widely  available 

Updates  of  trends  in  typical  use  of  measurement  in  software  &  systems 
engineering 

To  help  projects  &  organizations  judge  their  progress  relative  to  others 

•  But  there  also  will  be  a  continuing  need  to  track  qualitative  as  well  as 
quantitative  descriptions  about  the  quality  &  frequency  of  use  of 
measurement  in  our  field 
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2nd  Annual  SEI  Measurement  Practice  Survey 


New  this  year 

•  Screening  question  to  identify  respondents  whose  organizations  develop 
software  but  rarely  if  ever  do  measurement 

•  Questions  about 

Resources  &  infrastructure  devoted  to  measurement 

Practices  to  ensure  data  quality  &  integrity 

Value  added  by  doing  measurement 

The  kinds  of  measures  used  by  the  responding  organizations 

Among  other  things,  these  questions  allow  us  to  make  some  useful 
comparisons  by  CMMI  maturity  level 
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Trends  over  Time 


1st  survey  described  at  last  year's  CMMI  technology  Conference 
Similar  results  this  year 

•  Moderately  strong  relationships  exist  when  comparing  the  replies  of 
respondents  based  on: 

Management  versus  staff  roles 

—  Industry  versus  government  organizations 

The  United  States  versus  other  countries 

Organization  size 

But  that’s  a  topic  for  another  time 
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CMMI  Measurement  Capabilities  &  Performance 
Outcomes 


Today’s  focus 

•  Provide  evidence  about  the  circumstances  under  which  measurement 
capabilities  and  performance  outcomes  are  likely  to  vary 

•  As  a  consequence  of  achieving  higher  levels  of  CMMI  maturity 

Most  differences  are  consistent  with  expectations  based  on  CMMI 

•  Which  provides  confidence  in  the  validity  of  the  model  structure  &  content 

However,  the  results  also  highlight  areas  where  sometimes  considerable 
room  for  improvement  remains 

•  Even  at  maturity  levels  4  and  5 

•  For  example 

A  rather  strong  overall  relationship  exists  between  maturity  level  &  use 
of  measures  about  quality  attributes 

Little  attention  to  quality  attributes  at  the  lower  maturity  levels 

Yet,  almost  half  of  maturity  level  4  &  5  respondents’  organizations  track 

quality  attributes  only  occasionally  at  best 
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The  Sample 


Random  sample  of  SEI  customers 

•  944  valid  email  invitations  to  participate 

Data  collected  20  February  through  10  April  2007 

•  Two  reminders 


Response  rate 


•  41%  completed  all  or  part  of  the  questionnaire 

•  N  =  384 

•  Individual  questions  answered  by  75-97%  of  respondents 

~29  -  39%  of  the  sample  invitees 
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Today’s  Talk 


Purpose  &  scope  of  the  survey 
Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 
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Role  in  the  Organization 


N  =  366 


9%  4% 


Software  Engineering  Institute 


13% 


□  Executive 

■  Program  manager 

□  Project  manager 

□  Engineer 

□  Programmer 

■  Analyst 

□  Other 
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Who  are  the  others? 


=  8%  of  all 
those 

responding 


N  =  155 


9% 


Software  Engineering  Institute 


□  Quality 

■  Process 

□  Process  +  Quality 

□  Consultant 

□  Management 

■  Other  Others 
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And  who  are  the  other  others? 


Process  +  Measurement  3 

Measurement  Specialist  1 

Process  +  Quality  >  6 

+  Measurement  +  Training  1 

Quality  +  Process 

+  Measurement  1  , 

Training  6 

Architect  4 

Security  2 

Testing  2 

N  =  31 


One  each: 

•  Administrative  support 

•  Coach 

•  Consultant  +  researcher 

•  Engineering  Manager  +  Process 

•  Process  +  Project  engineer 

•  Program  /  team  lead 

•  Program  manager  +  Quality  + 
Process 

•  Project  manager  +  Quality 

•  Project  manager  +  Engineer 

•  Not  specified 
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Sector 


7%  4% 


N  =  366 


13% 


— Software  Engineering  Institute 


37% 


□  Commercial  shrink-wrap 

■  Custom  software 
development 

□  In-house  or  proprietary 

□  Defense  contractor 

□  Other  government 
contractor 

■  Defense  or  military 
organization 

□  Other  government  agency 

□  Consultancy 

■  Other 
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Country 


N  =  363 


12% 


— Software  Engineering  Institute 


□  United  States 

■  India 

□  Japan 

□  France 

□  Germany 

■  United  Kingdom 

□  Canada 

□  Netherlands 

■  All  others 
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FTE  Staff 


Percent 


N  =  364 


50  or  51-100  101-200  201-500  501-2000  More  than 

fewer  2000 
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Maturity  level 


Percent 


Level  1  Level  2  Level  3  Level  4  Level  5  Don't 
N  =  365  Know 
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Differences  by  Maturity  Level: 

Use  of  Measurement  in  the  Organization 


ML1&DK  ML2  ML3  ML4&5 

N  =  1 51  N  =  84  N  =  59  N  =  71 


Gamma  =  .73  p  <  .0001 
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Interpreting  the  results: 

The  Respondents’  Measurement  Roles 


ML1&DK  ML2  ML3  ML4&5 

N  =  1 51  N  =  84  N  =  59  N  =  70 


p  =  .04 


=  — Software  Engineering  Institute  CarnegieMellon 


i 

i 


Other 

Neither 


Both 


User 

Provider 
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Today’s  Talk 


Purpose  &  scope  of  the  survey 
Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 


_  Understanding  CMMI  Measurement 

Software  Engineering  Institute  Carnegie  Mellon  SETloidenson,  n  November  2007 

©2007  Carnegie  Mellon  University 


How  Measurement  Work  is  Staffed 


3%,  1  %,  2%  &  3%  respectively 


[ 

]  Other 

L 

e 

Don’t  know 

Organization 

wide  group 

Project 

level 

i  A  few  key 
i  experts 

ML1&DK  ML2  ML3  ML4&5 

N  =  78  N  =  60  N  =  58  N  =  60 


p  <  .006 
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Earmarked  Budgets  for  Measurement 


ML1&DK  ML2  ML3  ML4&5 

N  =  76  N  =  68  N  =  50  N  =  61 


Don’t 

know 


No 


Yes 


p  <  .0001 
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Availability  of  Qualified  Measurement  Staff 


Rarely,  never 

&  don’t  know 

Half  the  time  & 

occasionally 

!!  Almost  always 
j  &  frequently 

ML1&DK  ML2  ML3  ML4&5 

N  =  76  N  =  65  N  =  50  N  =  61 


Gamma  =  .44  p  <  .0001 
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Similar  Results 


For: 

•  Automated  measurement  support  for  data  collection,  data  management, 
data  analysis  &  reporting 

•  Use  of  commercial  measurement  packages  &  tools 

•  Existence  of  common,  integrated  organizational  measurement  repositories 

•  Availability  of  measurement  related  training 

Proportions  sometimes  vary  across  the  distributions. 

But  there  are  consistent  differences  by  maturity  level. 
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Today’s  Talk 


Purpose  &  scope  of  the  survey 
Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 
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Effects  of  Measurement  on  the  Organizations-, 


Better  Project  Performance 


Better  Product  Quality 


4% 


...........  . 


/O 


ML1&DK  ML2  ML3  ML4&5 
N  =  74  N  =  60  N  =  50  N  =  56 


Rare,  never, 
worse,  DK 
or  NA 


Half  time  or 
on  occasion 


Always  or 
frequently 


ML1&DK  ML2  ML3  ML4&5 
N  =  74  N  =  60  N  =  50  N  =  56 


Gamma  =  .41  p<.0001  Gamma  =  .34  p  <  .0002 
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Effects  of  Measurement  on  the  Organizations2 


Better  Tactical  Decisions 


Better  Strategic  Decisions 


ML1&DK  ML2  ML3  ML4&5 
N  =  74  N  =  59  N  =  50  N  =  56 


Rare,  never, 
worse,  DK 
or  NA 


Half  time  or 
on  occasion 


Always  or 
frequently 


ML1&DK  ML2  ML3  ML4&5 
N  =  74  N  =  59  N  =  49  N  =  55 


Gamma  =  .35  p  =  .0001  Gamma  =  .31  p  =  .0008 
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Today’s  Talk 


Purpose  &  scope  of  the  survey 
Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 
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Project  &  Organizational  Measurement  Results 
Reported-, 


Cost  Performance 


Schedule  Performance 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  55  N  =  45  N  =  51 


Rarely, 
never,  DK, 
or  NA 


Occasionally 


Frequently 


Regularly 


4% 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  56  N  =  44  N  =  51 


Gamma  =  .25  p  <  .03  Gamma  =  .37  p  =  .0006 
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Project  &  Organizational  Measurement  Results 
Reported2 


Business  Growth  & 
Profitability 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  55  N  =  45  N  =  51 


1 


Rarely, 
never,  DK, 
or  NA 


Occasionally 


Frequently 

Regularly 


Gamma  =  .20  p  =  .2244 
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Product  &  Quality  Measurement  Results  Reported-, 


Requirements  / 
Architectures 


Quality  Attributes 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  55  N  =  45  N  =  51 


Rarely, 
never,  DK, 
or  NA 


Occasionally 


Frequently 


Regularly 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  55  N  =  45  N  =  52 


Gamma  =  .37  p  =  .0002  Gamma  =  .32  p  <  .008 


_  Understanding  CMMI  Measurement 

Software  Engineering  Institute  Carnegie  Mellon  SETloidenson,  n  November  2007 

©2007  Carnegie  Mellon  University 


30 


Product  &  Quality  Measurement  Results  Reported2 


Defect  Density 


Defect  Phase  Containment 


4% 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  56  N  =  45  N  =  52 


Rarely, 
never,  DK, 
or  NA 


Occasionally 


Frequently 


Regularly 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  56  N  =  45  N  =  51 


Gamma  =  .41  p  <  .0001  Gamma  =  .44  p  <  .0001 
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Product  &  Quality  Measurement  Results  Reported3 


Customer  Satisfaction 


ML1&DK  ML2  ML3  ML4&5 
N  =  70  N  =  56  N  =  45  N  =  52 


Rarely, 
never,  DK, 
or  NA 


Occasionally 


Frequently 


Regularly 


Gamma  =  .31  p  <  .005 
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Similar  Results 


For: 

•  Adherence  to  work  processes 

•  Effort  applied  to  task 

•  Estimation  accuracy 

•  Cycle  time 

Proportions  sometimes  vary  across  the  distributions. 
But  there  are  consistent  differences  by  maturity  level. 
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Today’s  Talk 


Purpose  &  scope  of  the  survey 
Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 
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Differences  by  Maturity  Level: 

Practices  to  Ensure  Data  Quality 


Statistical  estimates  of 


Checks  for  inconsistent 


measurement  error 


ML1&DK  ML2  ML3  ML4&5 
N  =  74  N  =  56  N  =  47  N  =  51 


interpretation 

6% 


Rarely, 
never,  or 
DK 


Half  time  or 
on  occasion 


Always  or 
frequently 


ML1&DK  ML2  ML3  ML4&5 


N  =  74  N  =  57  N  =  48  N  =  50 


Gamma  =  .44  p<.0001  Gamma  =  .44  p  <  .0001 
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Differences  by  Maturity  Level: 

Practices  to  Ensure  Data  Quality 


Checks  for  unusual 

distribution  patterns 

2% 


ML1&DK  ML2  ML3  ML4&5 
N  =  74  N  =  58  N  =  48  N  =  51 


Rarely, 
never,  or 
DK 


Half  time  or 
on  occasion 


Always  or 
frequently 


Gamma  =  .46  p<.0001 
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Similar  Results 


For: 

•  Out  of  range  &  illegal  values  ...  Number  &  distribution  of  missing  data 

•  Missing  data  not  treated  as  zero  ...  Precision  &  accuracy  tests 

•  Other  aspects  of  alignment  &  coordination  of  measurement  activities 

Understandable  &  consistent  measurement  definitions 

Understandable  &  interpretable  measurement  results 

Use  of  “standard”  measurement  methods 

Measurable  product  &  service  criteria 

Measurement  used  to  understand  product  &  service  quality 

Documented  data  collection  process 

Documented  process  for  reporting  results 

Corrective  action  taken  when  thresholds  exceeded 

Understands  purposes  of  the  data  collected/reported 

Proportions  sometimes  vary  across  the  distributions. 

But  there  are  consistent  differences  by  maturity  level. 
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Today’s  Talk 


Purpose  &  scope  of  the  survey 
Results 

•  The  respondents  &  their  organizations 

•  Measurement  resources  &  infrastructure 

•  Value  added  by  measurement 

•  Software  measures  used 

•  Data  quality  &  integrity 

•  Organizational  perspectives  on  software  measurement 

Summary,  lessons  learned  &  next  steps 
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Organizational  Perspectives 
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Decision  Making 


t  ML1&DK  ML2  ML3  ML4&5 
3%  N  =  1 02  N  =  61  N  =  41  N  =  53 


Hardly  at  All 


•v  Limited 


Some 

[]  Largely 
Entirely 


Gamma  =  .27  p  =  .0002 


Onerous  or  Burdensome 


4% 


ML1&DK  ML2  ML3  ML4&5 
N  =  110  N  =  67  N  =  45  N  =  52 


Gamma  =.17  p  <  .45 
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Similar  Results 


For: 

•  Stated  negatively 

Inappropriate  collection  &  use  of  data 
Resistance  to  “extra”  work 

•  Stated  positively 

Understandable  &  interpretable  results 
Data  collected  are  regularly  analyzed 
Measurement  an  integral  part  of  the  business 
Objective  results  highly  valued 

Once  again: 

•  Proportions  sometimes  vary  across  the  distributions. 

•  But  there  are  consistent  differences  by  maturity  level. 

Yet  resistance  to  measurement  still  exists  in  our  field. 

•  Even  in  high  maturity  organizations 
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Summary  of  Results 


Characteristic  differences  associated  with  CMMI  Maturity  level  achieved 

•  Measurement  capability  &  performance  outcomes 

•  Common  stair  step  pattern  up  the  maturity  levels 

•  Some  quite  substantial 

Still,  some  of  the  results  imply  room  for  improvement 

•  Sometimes  substantial  room 

Even  in  higher  maturity  organizations 

•  Although  the  expectations  for  quality  &  “goodness”  may  well  be  higher 
there  too 

•  Jim  Herbsleb  &  I  saw  a  similar  pattern  years  ago 

—  For  process  champions  versus  practitioners  &  managers 
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The  Future 


Relatively  little  data  yet  exist  for  meaningful  comparisons  among  software 
&  systems  engineering  projects  &  organizations 

•  Hence  tendency  to  cover  too  much  at  once  in  a  single  sample  survey 

Considering  variants  on  matrix  sampling  strategies  for  2008  survey 

•  Answer  only  a  subset  of  questions  ...  to  avoid  over-burdening  the 
respondents 

“State  of  the  practice”  can  refer  to  very  different  target  populations 

•  The  SEI  customer  base  ...  the  broader  software  &  systems  engineering 
community  ...  or  those  organizations  that  more  routinely  use  measurement? 

•  Of  course,  the  answer  depends  on  the  purposes  of  the  survey 
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Next  Steps 


Our  plans 

•  We  will  track  change  over  time  &  go  into  further  depth  about  focused  topics 
from  the  perspective  of  current  measurement  practitioners 

Considering  parallel  samples  for  2008 

•  A  short  set  of  questions  for  tracking  the  diffusion  of  measurement  through 
the  broader  software  &  systems  engineering  community 

•  Possible  focus  on  issues  faced  with  respect  to  the  adoption  &  use  of  high 
maturity  measurement  practices 

Also  fielding  a  survey  on  Program  Office  acquisition  capabilities  (early  2008) 

Of  course,  there  is  no  shortage  of  additional  topics  for  the  future 

•  In  the  SEI  series  or  in  those  that  we  hope  to  see  done  by  others 
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Thank  You  for  Your  Attention! 


Dennis  R.  Goldenson 

dq@sei.cmu.edu 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 
USA 
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